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Executive  Summary 


This  report  describes  a  series  of  ongoing  research  efforts  investigating  the  role  of  interdependence  in  the 
acquisition  of  Major  Defense  Acquisition  Programs  (MDAPs).  This  research  initiative  was  sponsored  by  the 
Office  of  the  Secretary  of  Defense  (OSD). 

The  overall  goal  of  the  research  was  to  identify,  quantify,  and  assess  the  degree  of  programmatic  and 
constructive  interdependence  and  to  assess  the  effects  of  interdependence  on  program  risk. 

This  paper  reports  the  results  of  five  research  studies  that  were  conducted  from  2004  to  2009. 

Study  1  explored  the  qualitative  factors  that  confound  program  cost  and  schedule  estimation. 

Study  2  employed  data-mining  and  statistical  analyses  to  determine  whether  Defense  Acquisition  Executive 
Summary  (DAES)  reports  and  Select  Acquisition  Reports  (SARs)  can  be  used  to  forecast  program 
performance.  An  interesting  result  from  this  study  is  that  there  was  no  evidence  that  such  indicators  are 
effective  in  predicting  program  breaches. 

Studies  3-5  employed  network  analysis  techniques  to  quantitatively  characterize  programmatic  and 
constructive  interdependencies  in  the  acquisition  enterprise.  These  last  three  studies  culminated  in  graphical 
models  that  relate  interdependence  and  program  cost. 

Four  critical  findings  were  revealed  by  this  research. 

1 .  Our  research  study  found  no  evidence  that  indicators  reported  within  DAES  reports  or  SARs  predict 
program  breach  events. 

2.  Limiting  the  definition  of  interdependence  to  programs  that  are  identified  and  funded  as  joint  programs  is 
insufficient.  All  programs  are  interdependent  to  some  degree,  and  therefore  interdependence-related  risk 
is  widespread,  regardless  of  service  affiliation  or  participation. 

3.  Continued  investment  in  the  gathering  of  objective,  authoritative  data  should  be  a  major  emphasis  across 
the  Department  of  Defense  (DoD).  The  consequence  of  failing  to  gather  and  analyze  these  important  data 
is  substantial  program  failure,  wasted  resources,  and  erosion  of  the  public  trust  in  the  integrity  and 
capability  of  the  DoD. 

4.  The  traditional  methods  of  analyzing  risk,  while  important,  need  to  be  supplemented  with  network 
analysis  techniques  to  reveal  the  true  scope  and  effects  of  programmatic  and  constructive 
interdependence.  Additional  investigation  into  the  methods  and  measures  that  can  reveal  critical 
interdependencies  is  clearly  warranted. 
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Abstract 


The  challenges  program  managers  encounter  in  attempting  to  deliver  programs  on  time  and  on  budget  are  well 
substantiated.  A  significant  driver  of  the  turbulence  experienced  by  acquisition  programs  today  is  the 
transformation  to  joint  capabilities.  This  report  describes  a  series  of  ongoing  research  efforts,  sponsored  by  the 
Office  of  the  Secretary  of  Defense  (OSD),  that  investigated  the  role  of  interdependence  in  the  acquisition  of 
major  defense  acquisition  programs. 

The  overall  goal  of  the  research  was  to  identify,  quantify,  and  assess  the  degree  of  programmatic  and 
constructive  interdependence  and  to  assess  the  effects  of  interdependence  on  program  risk.  A  number  of 
important  findings  and  noteworthy  insights  were  discovered  as  programs  were  examined  in  light  of  their 
interdependencies  with  other  programs.  The  results  indicate  that  an  expanded  definition  of  interdependencies 
along  with  the  incorporation  of  network  analysis  tools  may  provide  important  insights  into  program  performance 
in  a  joint  capability  arena. 
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1  Introduction 


1.1  Background 

The  challenges  program  managers  encounter  in  attempting  to  deliver  programs  on  time  and  on  budget  are  well 
substantiated.  A  significant  driver  of  the  turbulence  experienced  by  acquisition  programs  today  is  the 
transformation  to  joint  capabilities.  With  the  end  of  the  Cold  War,  and  the  erosion  of  the  relatively  stable  East- 
versus-West  power  structure,  multi-lateral,  state-  and  non-state-affiliated  threats  emerged.  Strategic  and 
operational  advantage  shifted  from  mass  and  firepower  to  agility  and  precision  [CJCS  2000]. 

This  need  for  integrated  joint  operations  triggered  corresponding  changes  in  acquisition  policy  and 
organizational  processes.  A  variety  of  initiatives  have  been  enacted  that  are  intended  to  rectify  the  situation. 
These  include  the  Grace  Commission  [Grace  1984],  the  Packard  Commission  [Packard  1986],  the  Goldwater- 
Nichols  Act  of  1986  [Lederman  1999],  the  Defense  Acquisition  Performance  Assessment  [Kadish  2006],  and 
multiple  Defense  Science  Boards  and  GAO  reports.  In  addition,  pending  legislation  intends  to  impose  further 
restructuring  of  defense  acquisition  processes,  organizations,  and  the  workforce  [Doyle  2009]. 

Secretary  of  Defense  Donald  Rumsfeld’s  call  for  transformation  drove  new  organizational  performance  goals 
that  stressed  adaptive  planning,  accelerated  acquisition  cycles,  output-based  management,  and  a  reformed 
analytic  support  agenda.  Continuing  on  this  theme,  Secretary  of  Defense  Robert  Gates  has  encouraged 
“jointness.”  Documents  such  as  the  Capstone  Concept  for  Joint  Operations  [CJCS  2009]  aim  to  both  inform 
and  equip  military  leaders  with  the  ability  to  meet  future  threats.  In  recognition  that  integration  doesn’t  “just 
happen,”  the  DoD  created  linkages  between  joint  operational  capabilities  and  the  DoD  acquisition  system  such 
as  those  described  through  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  [CJCSI 2005] 
and  the  various  DoD  interoperability  standards  [CJCSI  2008,  DoDD  2004]. 

However,  despite  vigorous  acquisition  reform,  oversight,  and  scrutiny,  cost  overruns  and  schedule  delays 
remain  unacceptably  high.  Fundamentally  similar  issues  continue  to  plague  the  DoD  acquisition  process.  This 
leads  us  to  ask  whether  we  have  identified  the  root  of  the  underlying  problem  that  drives  the  observed 
behaviors. 

1.2  The  Problem 

Adjusting  to  the  needs  of  joint  capabilities  has  not  been  an  easy  task,  and  organizational  theorists  do  not  have 
much  empirical  advice  to  offer  [Provan  2007,  Olsen  2005,  Meier  2008,  Agranoff  2003]. 

The  fact  remains  that  the  transformational  mandate  of  top  DoD  leadership  to  achieve  joint  capabilities  is 
superimposed  on  a  foundation  reflecting  the  traditional  service-led,  program-centric  acquisition  paradigm. 
Experience  over  the  past  decade  demonstrates  that  the  institutional  focus  on  the  program  as  the  principal 
management  mechanism  tends  to  blunt  awareness  of  inter-programmatic  issues,  such  as  integration  and 
interoperability,  particularly  when  these  issues  cross  service  or  other  organizational  boundaries. 

The  effects  of  this  discontinuity  have  not  been  fully  articulated,  but  it  is  striking  that  in  an  era  where  jointness, 
interoperability,  and  capabilities-based  focus  are  dominant  themes,  the  acquisition  process  is  still  measured  and 
evaluated  primarily  from  a  program-centric  perspective  [GAO  2009].  Therefore,  it  is  perhaps  not  surprising  that  the 
response  to  the  perceived  shortfalls  in  the  acquisition  process  is  addressed  in  a  largely  program-centric  way. 
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For  the  acquisition  community  and  its  stakeholders,  a  fundamental  paradigm  shift  would  be  the  recognition 
that  individual  programs  are  not  isolated  and  sovereign  but  are  inextricably  interdependent  with  other 
programs  and  multiple  external  factors  that  resist  program-specific,  control-based  strategies.  For  improvement 
efforts  to  be  effective  where  prior  efforts  have  failed,  they  must  develop  different  perspectives  to  augment  the 
prevailing  program-centric  paradigm  and  reveal  the  “hidden”  drivers  of  program  behaviors.  By  and  large,  the 
study  of  interdependence  and  its  effects  on  government  programs  remains  in  its  infancy.  Clearly,  further 
applied  research  in  systems  development  and  acquisition  management  of  integrated  capabilities  is  necessary. 


1.3  Our  Hypothesis 

The  central  hypothesis  that  steered  our  research  efforts  is  that  interdependence  is  a  significant  contributor  to 
the  behavior  and  performance  of  acquisition  programs. 

As  illustrated  in  Figure  1,  operational  demand  for  joint  capabilities  establishes  the  need  for  interoperability. 
Interoperability,  by  definition,  entails  interactions  among  two  or  more  entities — and  in  the  implementation  of 
these  interactions,  interdependencies  are  necessarily  created.  These  resultant  interdependencies  must  be 
addressed  and  serviced  through  integration  activities.  It  is  the  demands  of  these  interdependencies  that  impinge 
on  programs  by  driving  the  expenditure  of  effort  (integration  activities)  and  the  incurrence  of  cost  and  schedule 
overruns. 


Joint  capabilities 


Interoperability 


Interactions 


Interdependencies 


Integration  activities 


Establishes 
need  for 


Cause 


Figure  1:  Cause  and  Effect  Relationships 

We  propose  that  interdependence  can  be  examined  in  light  of  three  distinct  domains:  programmatic, 
constructive,  and  operational.1  The  three  dimensions  of  interdependence  are  characterized  in  Table  1. 


Use  of  these  constructs  (programmatic,  constructive,  and  operational  views)  was  adapted  from  the  System  of  Systems  Intero¬ 
perability  (SOSI)  model  [Morris  2004], 
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Table  1:  The  Three  Dimensions  of  Interdependence 


Domains  of 
Interdependence 

Description 

Programmatic 

Encompasses  the  activities  related  to  the  management  of  one  program 
in  the  context  of  other  programs  and  includes  all  acquisition,  financial, 
and  program  management  activities  that  govern  the  lifecycle  of  the 
systems  that  support  end-user  needs. 

Constructive 

This  dimension  is  the  nuts  and  bolts  of  what  we  commonly  think  of  as 
systems  and  software  engineering.  It  addresses  technologies  (and  the 
technical  activities  to  select  and  apply  them).  These  technologies 
commonly  include  shared  architectural  elements,  data  specifications, 
communication  protocols,  and  common  standards. 

Operational 

Refers  to  the  activities  related  to  the  actual  operation  of  a  system  by  the 
end  user  in  the  context  of  others  systems. 

As  illustrated  in  Figure  2,  interdependencies  are  driven  into  the  supporting  programmatic  structures  of  the 
acquisition  domain  through  corresponding  interdependencies  among  the  systems  to  be  developed,  in  response 
to  operational  user  needs. 


Figure  2:  Interdependencies  at  One  Dimension  Drive  Interdependencies  at  Another 

1.4  Research  Approach 

Flaving  postulated  that  interdependence  may  be  a  significant  contributor  to  cost  and  schedule  estimation  error, 
the  following  research  questions  were  proposed  and  investigated: 

•  What  are  the  qualitative  factors  that  contribute  to  system  cost  and  schedule  estimation  error? 

•  What  is  the  efficacy  of  using  program  oversight  information  to  mitigate  problems  and  predict  breaches? 

•  Flow  extensive  is  programmatic  interdependence? 

•  Is  programmatic  interdependence  increasing  over  time? 
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•  What  is  the  relationship  between  constructive  interdependence  and  a  program’s  development  resource 
demand? 

To  address  these  questions,  the  following  five  research  studies  listed  below  were  conducted  from  2004  to  2009. 


Study 

Description 

See  Page 

1 

Identification  of  Diagnostic  Risk  Indicators  Associated  with  Integration  and 
Interoperability 

5 

2 

Using  DAES-SARS  Information  for  Forecasting  Program  Performance 

7 

3 

Exploring  the  Extent  of  Programmatic  Interdependence 

15 

4 

Using  Network  Methods  to  Explore  the  Evolution  of  Acquisition  Program 
Interdependence  Over  Time 

19 

5 

Using  Network  Methods  to  Explore  the  Relationship  between  Systems 
Development  Interdependencies  and  Development  Resources 

25 

A  description  of  each  research  study  follows.  This  paper  concludes  with  a  summary  section  followed  by 
appendices  that  are  referred  to  within  some  of  the  study  descriptions. 
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2  Study  1 :  Identification  of  Diagnostic  Risk  Indicators  Associated 
with  Integration  and  Interoperability 


2.1  Research  Question 

What  are  the  qualitative  factors  that  contribute  to  system  cost  and  schedule  estimation  error? 

2.2  Background 

Having  postulated  that  interdependence  might  be  a  significant  contributor  to  the  behavior  and  performance  of 
acquisition  programs,  we  initially  examined  the  qualitative  factors  that  we  believe  contribute  to  errors 
associated  with  system  cost  and  schedule  estimation.  This  led  to  identification  of  a  set  of  categorized 
diagnostic  risk  factors  that  we  believe  impact  system  cost  and  schedule  performance.  As  some  portion  of 
program  risk  often  matures  into  program  problems,  we  assume  a  corresponding  relationship  to  program  cost 
and  schedule  estimation  error. 

2.3  Method 

The  approach  was  to  conduct  a  literature  search,  review  survey  information,  and  to  interview  subject  matter 
experts. 

2.4  Results  and  Discussion 

A  list  of  risk  indicators  was  synthesized  from  research  and  then  grouped  into  four  categories: 

1 .  Missing  Requirements  (constructive  issues) 

2.  Organizational  &  Institutional  Obstacles  (programmatic  issues) 

3.  Lifecycle  Sustainment  (operational  issues) 

4.  Team  Performance  (the  ability  of  teams  to  address  constructive,  programmatic,  and  operational  issues 
that  arise) 

The  categorized  risk  indicators  are  presented  in  Appendix  A. 

These  results  are  presented  as  “rules  of  thumb”  guidance  that  can  alert  an  analyst  or  manager  about  potential 
program  risk. 

Each  risk  statement  is  accompanied  by  a  plus  (+)  or  minus  (-)  sign  to  indicate  the  types  of  characteristics  that 
we  believe  mitigate  (+)  or  exacerbate  (-)  the  associated  risk  of  a  particular  heuristic.  The  guidance  does  not 
provide  a  scoring  algorithm  for  the  set  of  heuristics. 

We  believe  this  guidance  will  resonate  with  experienced  software-oriented  program  management  and  that 
these  findings  are  relevant  to  broader  topic  areas  in  systems  engineering. 
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3  Study  2:  Using  DAES-SARS  Information  for  Forecasting  Program 
Performance 


3.1  Research  Question 

What  is  the  efficacy  of  using  program  oversight  information  to  mitigate  problems  and  predict  breaches? 

3.2  Background 

This  study  examines  the  efficacy  of  using  the  program  oversight  information  reported  to  the  Office  of  the 
Secretary  of  Defense  (OSD)  acquisition  community  to  mitigate  problems  and  predict  breaches.  Project 
managers  submit  required  information  to  acquisition  organizations  in  the  U.S.  Department  of  Defense  (DoD), 
which  is  then  collected  in  the  Defense  Acquisition  Executive  Summary  (DAES),  published  on  a  quarterly 
basis,  and  the  Select  Acquisition  Report  (SAR),  published  on  an  annual  basis.  Using  DAES  reports  and  SARs, 
program  development  issues  can  be  examined  on  a  quarterly  basis  using  standardized  variables  to  indicate  risk. 
DAES  reports  also  form  a  major  component  of  acquisition  oversight,  with  appropriate  defense  agencies  adding 
their  assessment  of  the  risk  indicators  to  the  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR)  database. 

In  this  study,  we  examined  a  single  major  defense  acquisition  program  (MDAP)  that  suffered  multiple  breach 
events  over  a  12-year  period.  This  program — the  Multifunctional  Information  Distribution  System- Low 
Volume  Terminal  (MIDS-LVT)  program — began  as  a  concept  in  the  1970s,  became  an  international  military 
cause  celebre  during  the  1980s,  went  into  engineering  and  manufacturing  development  during  the  1990s,  and 
began  full-rate  production  after  2003.  The  current  program  plan  continues  with  production  and  deployment 
until  2012.  This  study  uses  MIDS-LVT  data  from  1997-2006. 

The  MIDS-LVT  program  was  selected  as  our  target  program  due  to  its  longevity,  complexity,  the  availability 
of  unclassified  data,  and  the  many  SoS  characteristics  it  had  because  of  its  multinational  composition.  We 
examined  the  MIDS-LVT  program  to  see  how  well  information  generated  for  the  DAES  reports  corresponded 
to  cost  and  schedule  breaches  during  the  program  life  cycle. 

The  DAES  reports  and  SARs  are  the  major  sources  of  ongoing  information  regarding  program  performance 
reported  to  Pentagon  and  acquisition  authorities,  and  their  usefulness  relies  heavily  on  the  program  manager’s 
ability  to  communicate  relevant  and  useful  information.  DoD  5000.x  mandates  the  use  of  particular  fields  for 
these  reports,  which  leads  to  consistency  of  data  over  time.  In  an  attempt  to  stay  ahead  of  the  game  and 
manage  emerging  issues  that  impact  cost  and  schedule — particularly  breaches — the  Program  Assessment 
Indicators  of  the  DAES  reports  require  explanations  by  the  program  manager  in  the  following  categories: 

•  performance  characteristics 

•  test  and  evaluation 

•  logistics  requirements  and  readiness  objectives 

•  cost 

•  funding 

•  schedule 
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contracts 


•  production 

•  management  structure 

•  interoperability 

Each  indicator  is  assigned  a  ranking  of  green,  green  advisory,  yellow,  yellow  advisory,  or  red,  depending  on 
the  assessed  severity  of  issues  in  each  category.  If  sufficient  information  was  encapsulated  by  the  G-Y-R 
evaluations,  we  hypothesized  that  there  would  be  a  correlation  between  the  assessment  scores  and  the 
occurrence  of  breaches. 

3.3  Method 

Information  was  extracted  from  all  available  MIDS-LVT  DAES  reports  regarding  the  program  assessment 
indicators.  We  examined  the  data  to  determine  whether  the  assessment  indicators  could  predict  the  occurrence 
of  a  breach. 

Because  the  breach  is  a  binary  variable,  logistic  regression  was  chosen  as  the  analytical  technique.  In  this  case, 
the  logistic  regression  equation  is  modeling  the  probability  of  a  breach  occurring  (or  not  occurring)  based  on 
the  ratings  of  the  assessment  indicators. 

In  order  to  perform  the  analysis,  numbers  were  assigned  to  the  color  scale  used  in  program  reports.  The 
mapping  between  the  color  scale  and  the  scale  used  for  analysis  is  as  follows: 

•  1  =  (G)  green 

•  2  =  (GA)  green  advisory 

•  3  =  (Y)  yellow 

•  4  =  (YA)  yellow  advisory 

•  5  =  (R)  red2 

3.4  Results  and  Discussion 

Information  extracted  from  all  available  MIDS-LVT  DAES  reports  regarding  the  program  assessment  indictors 
is  shown  in  Table  2. 3 


2  The  DAES  guidance  document  at  https://acc.dau. mil/GetAttachment.aspx?id=24422&pname=file&aid=2852  instructs  that  the 
advisory  codes  can  indicate  a  worsening  or  improving  status.  This  situation  raises  the  possibility  of  yellow  advisory  being  used 
to  indicate  a  transition  between  green  and  yellow.  The  reading  of  the  actual  descriptions,  however,  indicates  that  yellow  advi¬ 
sory  is  operationally  used  as  the  transition  between  yellow  and  red. 

3  Many  of  the  organization  name  designations  (e.g.,  in  column  1  of  Table  2)  have  changed  since  this  research  was  conducted. 


8  |  CMU/SEI-201 0-TR-024 


Table  2:  DAES,  DAES  Assessment  Indicators,  and  SAR  Breach  Data 


Nov 

97 

Feb 

98 

Nov 

98 

Feb 

99 

Jun 

99 

Nov 

99 

Feb 

00 

Aug 

00 

Nov 

00 

Feb 

01 

May 

01 

Aug 

01 

Nov 

01 

Feb 

02 

May 

02 

Aug 

02 

Nov 

02 

Feb 

03 

May 

03 

Aug 

03 

Nov 

03 

Feb 

04 

May 

04 

Aug 

04 

Nov 

04 

Feb 

05 

May 

05 

Aug 

05 

Nov 

05 

Feb 

06 

May 

06 

Baselines  -  Original:  Mar  8, 
1994 

14 

Jun 

19 

Sep 

18 

Jul 

14 

Jun 

13 

Mar 

18 

Jun 

22 

Mar 

Breach1,2 

Dec  R,P 

Dec 

S 

S 

Dec 

S 

Dec 

R,P 

Dec 

S 

Dec 

S,P 

Jun 

S 

Dec 

R 

Performance  Characteristics 

G 

G 

G 

G 

G 

G 

NR 

NR 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

YA 

NR 

Y 

NR 

GA 

GA 

GA 

YA 

Y 

Y 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

G 

G 

YA 

Y 

GA 

Y 

GA 

GA 

GA 

GA 

G 

G 

G 

G 

GA 

GA 

GA 

G 

Test  &  Evaluation 

GA 

GA 

Y 

Y 

Y 

GA 

NR 

NR 

YA 

YA 

YA 

YA 

YA 

Y 

YA 

Y 

Y 

Y 

Y 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

Y 

YA 

YA 

YA 

YA 

YA 

YA 

Y 

Y 

Y 

Y 

YA 

Y 

GA 

GA 

GA 

G 

GA 

GA 

GA 

GA 

GA 

GA 

YA 

YA 

YA 

YA 

YA 

YA 

YA 

GA 

GA 

GA 

GA 

G 

GA 

GA 

GA 

GA 

GA 

Logistics  Requirements  & 
Readiness 

G 

G 

GA 

GA 

GA 

GA 

NR 

NR 

G 

G 

GA 

GA 

Y 

Y 

G 

G 

G 

G 

Y 

G 

G 

G 

GA 

Y 

Y 

G 

G 

G 

G 

Y 

GA 

G 

G 

G 

G 

G 

G 

G 

GA 

YA 

YA 

Cost 

G 

G 

Y 

Y 

Y 

Y 

NR 

NR 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

GA 

GA 

G 

G 

G 

G 

GA 

GA 

NR 

G 

G 

G 

G 

G 

NR 

G 

G 

G 

G 

G 

G 

GA 

G 

GA 

GA 

GA 

GA 

Y 

Y 

YA 

YA 

GA 

NR 

NR 

YA 

YA 

YA 

NR 

YA 

Y 

NR 

NR 

Funding 

GA 

GA 

Y 

G 

G 

G 

G 

GA 

GA 

GA 

GA 

Y 

Y 

Y 

Y 

G 

Y 

G 

G 

G 

G 

G 

GA 

_ 

R 

Y 

G 

G 

G 

G 

G 

G 

GA 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

Y 

G 

G 

G 

G 

GA 

_ 

R 

Y 

G 

GA 

GA 

GA 

GA 

GA 

Y 

Schedule 

Y 

Y 

■  GA 

NR 

NR 

Y 

Y 

GA 

GA 

GA 

GA 

GA 

GA 

GA 

Y 

YA 

G 

Y 

YA 

YA 

YA 

G 

G 

G 

G 

GA 

GA 

GA 

GA 

Y 

Y 

Y 

Y 

GA 

Y 

YA 

GA 

Y 

YA 

Y 

Y 

Y 

YA 

YA 

G 

GA 

G 

G 

GA 

G 

Y 

Y 

Y 

GA 

GA 

GA 

Y 

Contracts 

G 

GA 

GA 

Y 

YA 

Y 

NR 

NR 

GA 

Y 

Y 

Y 

Y 

GA 

GA 

GA 

GA 

G 

G 

G 

G 

G 

G 

Y 

G 

G 

G 

G 

G 

G 

GA 

GA 

G 

Y 

GA 

Y 

Y 

Y 

Y 

G 

G 

G 

G 

G 

G 

YA 

YA 

Y 

Y 

Y 

Y 

Y 

GA 

GA 

Production 

G 

G 

Y 

Y 

Y 

GA 

NR 

G 

GA 

Y 

Y 

Y 

Y 

GA 

GA 

GA 

GA 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

Y 

Y 

GA 

GA 

GA 

GA 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

GA 

G 

GA 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Management  Structure 

G 

G 

G 

Y 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

■  GA 

G 

G 

G 

G 

G 

GA 

Y 

Y 

Y 

Y 

Y 

G 

G 

G 

G 

G 

G 

G 

G 

Interoperability 

NR 

NR 

G 

G 

G 

GA 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

NR 

GA 

G 

G 

GA 

GA 

G 

G 

G 

G 

GA 

GA 

GA 

Y 

Y 

GA 

GA 

GA 

GA 

Y 

YA 

G 

G 

GA 

G 

GA 

GA 

GA 

G 

G 

GA 

GA 

GA 

GA 

GA 

G 

G 

G 

G 

G 

G 

G 

*  The  DAES  report  data  (rows  with  bold  headers  and  PM  suffixes)  are  usually  dated  one 
month  earlier  than  OSD  DAES  assessment  data  (rows  below  each  bolded  header  row). 
For  convenience,  we  have  grouped  by  DAES  assessment  dates. 
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1  Breaches  identified  from  SARs 

2  Breach  Codes:  S  =  Schedule 

R  =  RDT&E  (Cost) 

P  =  Procurement  (Cost) 

NR  =  explicitly  marked  as  not  rated 


Nine  program  breach  events  occurred  within  the  time  period  of  the  31  sampled  reports.4  However,  there  is  no 
apparent  visual  relationship  between  any  specific  indicator  and  any  breach  identified  by  the  SARS  reports. 

The  logistic  regression  for  using  all  the  indicators  to  predict  breaches  is  shown  in  Table  3. 

Table  3:  Combined  Indicators 


Predictor  (Indicators) 

Coef 

SE  Coef 

P 

Constant 

79.075 

53534.1 

0.999 

Performance  Characteristics 

-22.242 

10483.4 

0.998 

Test  &  Evaluation 

-38.906 

15911.3 

0.998 

Logistics  Requirements  &  Readiness 

-0.566 

1.5 

0.717 

Cost 

-19.570 

16374.3 

0.999 

Funding 

1.086 

1.7 

0.518 

Schedule 

-0.762 

2.3 

0.741 

Contracts 

21.648 

20869.5 

0.999 

Production 

-2.635 

24078.8 

1.000 

Interoperability 

42.520 

39999.2 

0.999 

These  results  indicate  no  relationship  (in  all  cases  P»0.05,  indicating  no  significant  correlations)  when  using 
all  of  the  available  indicators  together  to  predict  the  occurrence  of  any  breach.  Similarly,  no  significant  results 
were  found  when  correlating  against  each  category  of  breach  and  against  project  baseline  dates  (the  second 
row  in  Table  2). 

The  use  of  a  lagged,  breach-dependent  variable  was  examined  based  on  the  notion  that  some  amount  of  time 
might  expire  before  effects  became  noticeable  (a  maximum  of  six  months  was  considered).  Since  the  unlagged 
breach  variable  was  contemporaneous  with  the  indicators  reported  quarterly,  a  maximum  of  three  months 
could  have  expired. 

Upon  reduction  of  lags  in  the  sets  of  independent  variables  (i.e.,  the  indicators)  no  significant  models  emerged 
regardless  of  the  combinations  used.  Individual  predictors  were  then  evaluated. 

Table  4  displays  the  results  from  logistic  regressions  where  one  indicator  is  used  to  predict  any  breach  (cost  or 
schedule).  The  first  column  lists  the  indicator  used.5  The  second  column  lists  the  number  of  cases  available  for 
that  particular  equation.  The  number  of  cases  differs  due  to  OSD  agency  involvement — that  is,  different  OSD 
agencies  reviewed  different  indicators  at  different  parts  of  the  program  life  cycle.  The  third  column  lists  the 
p- value  associated  with  the  logistic  regression  coefficient  for  that  indicator.  The  fourth  column  indicates  the 
p-value  for  the  indicator  when  using  the  lagged  breach  variable.  Other  statistics  are  available  from  the  output, 
but  the  p-values  are  sufficient  for  us  to  make  a  determination  of  significance.  For  the  existence  of  a  significant 


Simultaneous  breaches  in  multiple  categories  are  counted  as  a  single  breach  event. 

Multiple  sources  provided  values  for  each  indicator  but  the  identity  of  these  sources  is  not  revealed  within  this  report. 
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relationship  between  the  indicator  and  the  occurrence  of  a  breach,  the  p-value  for  the  coefficient  must  be  less 
than  0.05. 

Notice  that  the  use  of  a  lagged  breach  variable  did  find  that  one  of  the  funding  indicators  was  significant  in 
predicting  a  breach.  We  have  no  explanation  of  this  occurrence  and  consider  it  an  artifact  to  be  compared  with 
cross-program  analyses. 

We  also  examined  whether  these  indicators  could  be  related  to  the  occurrence  of  new  cost  baselines,  which  are 
indicated  in  the  second  row  of  Table  2.  Again,  no  significant  relationships  were  detected. 

Table  4:  Isolated  Indicators 


Indicator 

#  of  cases 

p-value 

p-value 

lagged 

Performance  Characteristics 

28 

0.718 

0.415 

20 

1.000 

0.999 

18 

0.916 

0.821 

Test  &  Evaluation 

28 

0.708 

0.186 

24 

0.431 

0.119 

19 

0.398 

0.669 

Logistics  Requirements  &  Readiness 

27 

0.462 

0.830 

19 

0.465 

0.875 

11 

0.998 

0.348 

Cost 

28 

0.375 

0.114 

22 

0.738 

0.920 

5 

0.999 

0.999 

Funding 

28 

0.167 

0.019 

24 

0.471 

0.386 

6 

0.998 

0.304 

9 

0.525 

0.855 

Schedule 

28 

0.114 

0.096 

23 

0.057 

0.592 

9 

0.578 

0.998 

Contracts 

28 

0.334 

0.193 

15 

0.123 

0.726 

9 

1.000 

0.999 

Production 

29 

0.464 

0.747 

19,  19 

0.496 

0.414 

(insufficient  data) 

n/a 

n/a 

n/a 

(insufficient  data) 

n/a 

n/a 

n/a 
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Indicator 

#  of  cases 

p-value 

p-value 

Management  Structure 

28,  27 

0.999 

0.999 

21,  21 

0.930 

0.930 

Interoperability 

22,  22 

1.000 

0.999 

(insufficient  data) 

n/a,  5 

n/a 

0.999 

13,  12 

0.722 

0.194 

(insufficient  data) 

n/a 

n/a 

n/a 

(insufficient  data) 

21 

0.080 

0.882 

The  results  in  Table  5  show  the  lack  of  a  significant  relationship  between  the  schedule  program  assessment 
indicator  and  the  occurrence  of  schedule  breaches  only. 

Table  5:  Schedule  Breaches  Predicted  by  Schedule  Indicators 


Indicator  and  Sources 

#  of  cases 

p-value 

p-value 

lagged 

Schedule 

28 

0.148 

0.072 

23 

0.174 

0.416 

9 

0.999 

0.915 

These  results  confirm  the  visual  perception  that  there  is  no  significant  relationship  between  the  indicators  and 
the  occurrence  of  breaches.  Other  exploratory  analyses  conducted  using  the  actual  cost,  cost  variance,  and 
schedule  variance  also  showed  a  lack  of  statistically  significant  results.  See  Appendix  B  for  further  details  of 
the  analyses. 

As  we  attempted  to  find  evidence  of  systems -of-sy  stems -related  issues  across  a  broad  spectrum  of  programs, 
insurmountable  problems  arose  and  several  research  directions  were  abandoned.  Extensive  data  mining  was 
performed  on  several  data  sources  within  the  DoD,  such  as  the  Joint  C4I  Program  Assessment  Tool,  Cost 
Analysis  Improvement  Group  (CAIG)  historical  database,  and  the  DAES  repository.  Many  of  these  data 
sources  are  classified,  so  effort  was  expended  to  obfuscate  versions  of  classified  data  samples  to  evaluate  the 
feasibility  of  research  relevance.  This  approach  proved  to  be  very  labor  intensive  and  thus  cost  prohibitive. 

Many  of  the  integration  support  plans  that  were  examined  were  unclassified  and  appeared  to  be  a  good  source 
of  data.  It  later  became  apparent,  however,  that  mining  meaningful  data  from  these  reports  was  too  labor- 
intensive  and  subject  to  compromise  due  to  researcher  bias  in  interpreting  highly  variable  text-based  issue 
statements. 

It  became  evident  that  producing  a  large  cross-sectional  sample  to  study  would  be  beyond  our  resources.  We 
focused  instead  on  a  more  in-depth  study  of  a  single  program  to  try  to  find  data  that  was  easy  to  retrieve,  of 
consistent  quality,  and  relevant  to  our  research  goals.  We  hoped  this  study  would  give  us  a  justification  and 
mechanism  to  expand  the  data  set  across  many  programs. 
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Similar  findings  were  reported  previously  for  these  types  of  acquisition  reports.  In  the  Defense  Acquisition 
University  (DAU)  technical  report  Acquisition  Trend  Metrics  in  the  Department  of  Defense,  the  authors 
reviewed  34  programs  whose  engineering  and  manufacturing  development  phases  ended  between  1980  and 
1996.  They  concluded  that  “given  sufficient  detailed  milestones  in  program  documentation,  a  slip  in  early  test 
milestones  is  the  best  indicator  of  a  program  heading  into  trouble”  [Swank  2000].  Consistent  with  our  study  of 
the  MIDS-LVT  program,  this  DAU  study  found  that  there  was  not  sufficient  information  in  the  acquisition 
reports  to  address  technical  issues.  “The  conclusion:  it  is  not  possible  to  obtain  leading  indicators  of  a  program 
problem  in  the  technical  section  of  the  SAR.  Most  Acquisition  Category  (ACAT)  I  programs  are  so  complex 
that  not  even  a  very  good  engineer  analyst,  not  in  the  program  office,  could  follow  the  technical  evolution  of 
the  system”  [Swank  2000]. 

Gailey,  in  Predictive  Power  for  Program  Success  from  Engineering  and  Manufacturing  Development 
Performance  Trends,  examined  a  number  of  variables  collected  by  the  acquisition  process  and,  for  the  most 
part,  found  no  predictive  power  except  in  two  variables  [Gailey  2002].  Programs  that  used  a  “cost  plus 
incentive  fee”  contract  had  greater  success  than  programs  using  a  “cost  plus  award  fee,”  “fixed  price 
incentive,”  or  “firm  fixed  price”  contract.  Also,  contracts  that  had  no  competition  for  the 
demonstration/validation  phase  (dem/val)  had  better  success  in  engineering  and  manufacturing  development 
(EMD)  [Gailey  2002].  These  findings  correspond  well  with  our  view  of  the  MIDS-LVT  program. 

Another  source  of  DoD  cost  analysis  studies  that  used  S ARS/DAES  data  is  located  in  the  various  theses 
produced  at  the  Air  Force  Institute  of  Technology  (AFIT).  In  1996,  USAF  Captain  Gordon  found  no 
relationship  between  rebaselining  and  cost  overruns  [Gordon  1996].  Building  on  the  previous  studies  by  Sipple 
[Sipple  2002]  and  Genest  [Genest  2004],  Cross  took  a  two-stage  approach  to  statistical  modeling:  (1)  use  a 
logistic  regression  to  identify  whether  schedule  slips  occurred  and  (2)  use  multiple  regression  models  to 
predict  the  extent  of  schedule  variance  [Cross  2006].  In  these  models,  the  predictor  variables  accounted  for 
83%  of  the  schedule  variance.  A  more  recent  study  by  Foreman  resulted  in  five  regression  models  that 
predicted  cost  or  schedule  growth  with  adjusted  r2  above  0.80  [Foreman  2007].  These  theses  deserve  more 
study  as  they  use  different  numbers  of  cases  derived  from  SARS/DAES  reports  and  check  over  80  different 
variables  for  inclusion.  After  qualifying  their  methodologies,  most  of  these  models  use  less  than  40  cases  for 
prediction.  The  reasons  for  so  many  disqualifications  also  point  to  the  same  factors  we  mention  elsewhere  in 
this  report — unreported  or  missing  data,  changing  definitions  over  time,  and  lack  of  validation. 

Although  significant  time  and  effort  is  expended  reporting  and  reviewing  the  indicators  described  in  this 
report,  they  do  not  appear  to  have  any  value  in  predicting  breaches.  However,  we  can  report  that  a  wealth  of 
status  information  is  obtained  by  a  thorough  reading  of  the  assessments  and  of  the  DAES  reports.  In  particular, 
the  executive  summary  includes  a  section  called  “Significant  Developments  Since  Last  Report”  that  provides  a 
view  of  important  developments  and  changes  by  quarter.  Unfortunately,  such  information  was  not  useful  for 
predicting  breaches. 

The  DAES/SARs  reports  did  a  thorough  job  of  reporting  many  externalities  that  affected  the  MIDS  program. 
For  example,  at  one  point,  funding  by  one  of  the  NATO  partners  was  rejected  by  that  country’s  parliament, 
necessitating  a  temporary  bail-out  by  Spain  and  Italy  to  cover  costs.  Another  example  is  the  extensive,  ongoing 
Program  Manager  (PM)  discussions  about  planning  and  executing  the  various  tests  involved.  The  bulk  of  the 
technical  discussion  in  DAES/SARs  involved  testing,  which  is  an  external  factor  since  the  products  were  tested 
by  an  independent  DoD  testing  agency  that  had  no  ties  to  the  program. 
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In  general,  the  DAES  reports  appear  to  be  useful  for  keeping  track  of  program  issues  that  originate  externally 
to  the  technological  developments  required  by  the  project.  Detailed  explanations  are  included  for  many  of  the 
issues  that  affect  program  performance,  with  one  noticeable  omission:  there  is  little  information  about  the 
technology  development  issues  of  the  program.  Cost  and  contract  budget  data  are  reported,  but  no  connections 
to  system  developments  are  made  by  the  PM  or  Program  Executive  Office  (PEO). 
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4  Study  3:  Exploring  the  Extent  of  Programmatic  Interdependence 


4.1  Research  Question 

How  extensive  is  programmatic  interdependence? 

4.2  Background 

The  significance  of  interdependence  has  not  been  entirely  lost  on  the  acquisition  community.  Intuitively, 
program  managers  and  acquisition  staffs  have  understood  that  adding  the  joint  aspect  to  any  program  increases 
the  cost  and  complexity  of  programs  and  the  systems  they  acquire.  Awareness  of  the  potential  significance  of 
interdependence  has  emerged  in  the  form  of  the  Defense  Acquisition  Executive  Summary  (DAES)  review 
materials  developed  for  every  major  defense  acquisition  program  (MDAP)6.  In  this  summary,  programs  are 
required  to  identify  complementary  programs  which  impinge  upon  or  are  affected  by  the  subject  programs. 

4.3  Method 

DAES  interdependency  charts  for  2007  were  examined  and  analyzed  to  identify  program  interdependencies. 
The  interdependencies  were  graphically  rendered  as  links  between  programs  (nodes). 

4.4  Results  and  Discussion 

Figure  3  represents  the  program-to-program  interdependencies  identified  by  57 1  MDAP  programs  within  the 
DAES  interdependency  charts.  Note  that  1 194  program-to-program  interdependencies  are  identified  as 
represented  by  links. 

Figure  4  restricts  the  view  to  AC  AT  I  programs  only.  The  diagram  shows  65  programs  and  approximately  128 
interdependencies  represented  by  the  links  between  the  nodes. 


Every  designated  MDAP  prepares  an  annual  Defense  Acquisition  Executive  Summary  (DAES)  report  that  is  submitted  to 
OUSD  (AT&L).  Selected  programs  are  reviewed  at  a  quarterly  DAES  review.  Programs  so  selected  prepare  a  standard  briefing 
package  that  summarizes  program  status  and  issues.  Chart  5  of  that  package,  “Interrelationships,  Dependencies,  and  Syn¬ 
chronization  with  Complementary  Systems,”  provides  a  subjective  depiction  of  program  interdependencies. 
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Figure  3:  Mapping  of  Interdependencies  Among  all  DAES  Programs 
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Not  surprisingly,  an  analysis  of  the  DAES  interdependence  charts  shows  that  DoD  programs  are  indeed 
building  bridges  across  program  boundaries,  with  potentially  significant  effects  with  respect  to 
interdependence. 

Figure  3  and  Figure  4  provide  evidence  that  the  movement  to  joint  capabilities  is  well  underway,  and  is 
manifested  in  the  form  of  these  program-to-program  interdependencies.  What  this  view  does  not  provide, 
however,  is  insight  into  the  potential  benefits  and  consequences  of  widespread  interdependence.  In  other 
words,  if  interdependence  is  widespread,  does  this  necessarily  imply  a  significant  risk  for  programs  that 
possess  these  interdependencies? 

To  investigate  whether  interdependence  poses  increased  risk  to  programs,  we  looked  to  the  DoD’s  long 
experience  with  joint  program  acquisitions,  asserting  that  the  interdependencies  within  joint  programs 
represent  a  sub-class  of  interdependencies  within  the  general  set  of  interdependencies  identified  within  the 
DAES  charts.  DoD  programs  that  are  explicitly  identified  as  joint  efforts7  are  known  to  be  particularly 
challenging.  However,  this  recognition  does  not  prevent  these  programs  from  exceeding  cost  and  schedule 
thresholds  on  a  routine  basis.  The  fact  that  joint  programs  still  breach  more  than  twice  as  often  as  non-joint 
programs  suggests  that  the  complexity  related  to  interdependence  increases  above  informed  prior  expectations. 
Regardless  of  the  precise  reason  for  these  breaches,  the  evidence  suggests  that  the  complexity  of 
interdependencies — even  when  clearly  acknowledged  as  a  risk — appears  to  overwhelm  current  management 
capabilities.  Given  that  all  but  a  few  programs  are  not  born  as  “joint,”  systems  interdependencies  among  the 
armed  services  guarantee  interoperability  and  integration  issues. 

The  pervasiveness  of  interdependencies  also  has  significant  implications  for  programs  that  are  not  explicitly 
identified  as  joint  programs.  As  Figure  3  and  Figure  4  illustrate,  virtually  all  programs  are  interdependent — 
although  according  to  the  official  definition  of  joint  programs,  only  about  half  are  considered  to  be  joint.  As  a 
consequence,  programs  that  are  presumably  “non-joint”  (though  significantly  interdependent)  would 
experience  the  impact  of  interdependence  (i.e.,  cost  growth,  schedule  delay,  and  performance  shortfalls). 
However,  acquisition  management  would  not  anticipate  these  effects  because  without  the  insight  into 
interdependence,  programs  thus  affected  are  unlikely  to  properly  attribute  interdependence  as  a  root  cause  to 
these  effects.  Management  becomes  reactive  and  engages  in  firefighting  such  effects  as  they  emerge. 


The  Defense  Acquisition  Guidebook  defines  joint  acquisition  in  a  somewhat  narrow  sense.  It  states:  “A  joint  acquisition’  is  any 
acquisition  system,  subsystem,  component,  or  technology  program  with  a  strategy  that  includes  funding  by  more  than  one  DoD 
component  during  any  phase  of  a  system's  life  cycle”  [DoD  201 0]. 

Of  84  programs  in  2005,  45.4%  were  single-service  systems  and  53.6%  were  joint  systems  [Brown  2007], 
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5  Study  4:  Using  Network  Methods  to  Explore  the  Evolution  of 
Acquisition  Program  Interdependence  Over  Time 


5.1  Research  Question 

Has  programmatic  interdependence  increased  over  time? 

5.2  Background 

Study  3  explored  the  pervasiveness  of  programmatic  interdependence  within  a  single  year.  In  this  study,  we 
investigated  the  pattern  of  interdependence  of  programs  over  a  10-year  span  of  time  ( 1 997-2007). 9 

Organizational  theorists  have  long  recognized  that  the  exchange  of  resources  (e.g.,  goods  and  services) 
represent  a  source  of  interdependence.  Therefore,  we  examined  the  programmatic  dimension  of 
interdependence  through  funding  exchanges  between  programs  since  we  believe  that  these  exchanges  serve  as 
effective  proxies  for  the  exchange  of  goods  and  services  across  program  boundaries.  These  funding  exchanges 
were  tracked  by  means  of  the  DoD  program  element  (PE)  numbers.10 

5.3  Method 

The  data  set  used  in  this  study  was  comprised  of  Selected  Acquisition  Report  (SAR)  information  (collected 
during  from  1997-2007)  that  was  extracted  from  the  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR)  system. 

Funding  linkages  between  MDAPs  and  program  elements  (PEs)  were  explored  using  network  analysis  tools 
that  can  examine  and  report  characteristics  of  the  linkages  between  entities  (nodes). 


Data  for  FY2000  was  unavailable  and  therefore  not  included  in  the  analysis. 

10  The  DoD  uses  six-character  program  element  numbers  in  the  budget  process  to  identify  each  program. 
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5.4  Results  and  Discussion 


Table  6  lists  the  number  of  funding  linkages,  MDAPs,  and  PEs  for  each  of  the  ten  years  of  SAR  data  within 
the  data  set. 

Table  6:  Program  Elements  and  MDAP  Programs  1997-2007 


Year 

Number  of 
Linkages 

Number  of 

MDAPs 

Number  of 
Program 
Elements  (PEs) 

Ratio  of  PEs 
per  MDAP 

Number  of 
Links  per 
MDAP 

Number  of  Links 
per  PE 

1997 

82 

24 

80 

3.3 

3.4 

1.03 

1998 

96 

25 

92 

3.5 

3.7 

1.04 

1999 

87 

26 

84 

3.2 

3.3 

1.04 

2000 

Data  unavailable  for  FY2000 

2001 

113 

31 

104 

3.4 

3.6 

1.09 

2002 

116 

35 

105 

3.0 

3.3 

1.10 

2003 

117 

37 

106 

2.9 

3.2 

1.10 

2004 

135 

44 

120 

2.7 

3.1 

1.13 

2005 

159 

50 

135 

2.7 

3.2 

1.18 

2006 

257 

92 

218 

2.4 

2.8 

1.18 

2007 

319 

95 

257 

2.7 

3.4 

1.24 

Evidence  of  increasing  programmatic  interdependence  is  revealed  in  Figure  5,  which  depicts  the  number  of 
linkages  per  PE  over  time.  Throughout  its  life  cycle,  each  MDAP  will  exchange  funding  with  each  of  the  PEs 
that  it  is  associated  with,  including:  (1)  Research,  Development,  Test,  &  Evaluation  (RDT&E),  (2) 
Procurement,  and  (3)  Operating  and  Maintenance  (O&M)  appropriations.  However,  the  number  of  funding 
links  per  MDAP  is  greater  than  three  (for  each  year  of  its  life  cycle).  This  suggests  that  each  MDAP  is 
exchanging  resources  with  another  PE  at  some  time  in  its  life  cycle. 

This  dynamic  is  also  reflected  by  the  number  of  MDAPs  that  receive  funds  from  each  PE.  Each  PE  is  tied  to  a 
particular  appropriation  (e.g.,  RDT&E,  Procurement,  O&M),  and  each  PE  will  have  principally  one  program 
that  it  supports.  However,  the  average  number  of  MDAPs  linked  to  a  given  PE  has  grown  from  1.03  to  1.24. 
This  suggests  that  PEs  are  increasingly  providing  resources  to  more  than  their  primary  programs. 

Why  would  such  exchanges  take  place?  One  potential  explanation  is  that  these  links  reflect  exchanges  of 
goods  and  services  across  MDAP  boundaries.  Therefore  the  existence  of  these  funding  exchanges  reflects  the 
existence  of  program-to-program  interdependencies. 
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Figure  6  shows  the  pattern  of  resource  exchanges  (links)  between  PEs  and  MDAPs  as  of  1997.  Note  the  two 
clusters  that  are  revealed  in  the  diagram.  Both  clusters  show  that  only  two  MDAPs  share  common  program 
elements  in  1997.  Flowever,  by  2007,  the  relationships  had  grown  increasingly  complex,  as  illustrated  in 
Figure  7.  The  two  original  clusters  remained  essentially  the  same,  but  the  number  of  other  clusters,  and  their 
associated  link  densities,  increased  dramatically. 
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Figure  6:  Program  Clusters  as  of  1997 
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Figure  7:  Program  Clusters  as  of  2007 
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The  average  degree  for  a  network  is  defined  as  the  sum  total  of  all  node  linkages  divided  by  the  number  of 
nodes  within  the  network.  For  example,  in  Figure  8,  Node  A  has  three  links,  Node  B  has  four  links,  Node  C 
has  one  link,  Node  D  has  one  link,  Node  E  has  two  links,  and  Node  F  has  three  links.  Therefore,  the  average 
degree  (AD)  is  defined  as: 


AD 


Total  Node  Links  3  +  4  +  1  +  1  +  2  +  3 
#  of  Nodes  6 


Figure  8:  Example  -  Calculating  Average  Degree 

As  illustrated  in  Figure  9  the  average  degree  for  PE  to  MDAP  relationships  grew  from  0.2  during  1997  to  0.78 
during  2007 — an  approximate  four-fold  increase.  This  increase  occurred  even  though  the  average  number  of 
PEs  per  MDAP  did  not  change  over  the  10-year  timeframe. 


1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007 

Year 


Figure  9:  Average  Program  Network  Degree  by  Year 

The  growth  in  the  way  the  MDAPs  interconnect  through  their  respective  PEs  is  not  surprising  given  the 
movement  toward  joint  capabilities.  The  network  diagrams  support  the  hypothesis  that  the  complexity  of 
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acquisition  efforts  has  grown  along  with  increasing  emphasis  on  supporting  joint  capabilities.  Unfortunately 
this  increasing  complexity  is  hidden  by  the  program-centric  nature  of  the  SAR — only  by  compiling  individual 
observations  is  this  mosaic  of  interdependencies  and  corresponding  complexity  revealed. 

The  increasing  complexity  of  the  relationships  is  hidden  by  the  program-centric  nature  of  the  SAR.  However, 
by  compiling  individual  MDAP-PE  relationships,  this  mosaic  of  interdependencies  and  corresponding  com¬ 
plexity  is  revealed. 
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6  Study  5:  Using  Network  Methods  to  Explore  the  Relationship 
Between  Systems  Development  Interdependencies  and 
Development  Resources 


6.1  Research  Question 

What  is  the  relationship  between  constructive  interdependence  and  a  program’s  development  resource  demand? 

6.2  Background 

Constructive  interdependencies  are  a  result  of  the  technical  requirements  that  drive  the  development  of  a 
complex  system  to  meet  the  needs  of  the  end  user  (that  is,  the  operational  domain).  In  more  customary  terms 
for  the  DoD  acquisition  domain,  the  constructive  dimension  addresses  the  engineering  of  systems — the 
construction  of  system  components  and  the  integration  of  these  components  into  a  coherent  whole.  The 
artifacts  developed  within  the  constructive  domain  are  systems  engineering  artifacts  including  the  architectural 
depictions  that  describe  the  top-level  components  of  the  system  and  the  relationships  between  the  components. 

Recalling  our  fundamental  research  premise  that  interdependence  affects  the  effort  (and  thus  the  cost)  of 
developing  systems,  the  artifacts  most  likely  to  reveal  these  characteristics  are  those  that  explicitly  describe 
interdependencies.  The  Information  Support  Plan  (ISP)  is  such  an  artifact.  The  ISP  requires  the  program  to 
describe  the  information  support  requirements  the  system  in  question  receives  from  and  provides  to  external 
entities. 

The  interdependencies  described  in  the  ISP  are  limited  to  information  flows.  They  do  not  describe  such  things 
as  physical  interfaces,  electrical  power  requirements,  air  conditioning,  or  other  aspects  of  constructive 
interdependencies.  Furthermore  the  ISP  describes  these  information  flows  from  the  perspective  of  the  system 
itself,  rather  than  a  more  global  treatment  of  information  flows.  Nevertheless,  the  ISP  still  addresses 
interdependence  in  a  meaningful  and  consistent  way. 

With  the  advent  of  the  DoD  Architecture  Framework  (DoDAF),  a  standard  lexicon  and  semantics  for 
describing  constructive  interdependence  has  made  the  ISP  a  document  of  unique  significance  and  increasing 
utility  for  the  analysis  of  constructive  interdependence.  The  DoDAF  describes  entities  and  relationships 
through  a  number  of  “views”  which  are  designed  to  convey  information  in  a  way  that  is  relevant  to  a  particular 
constituency  in  terms  that  are  understandable  in  that  domain. 1 1 

One  of  the  principal  benefits  of  using  an  established  architecture  framework  like  DoDAF  is  to  enforce  linkages 
between  the  operational  and  the  constructive  domains  of  interdependence.  As  such,  the  resulting  integrated 
architecture  is  in  effect  a  systems  engineering  product  that  relates  operational  requirements  to  system  design 
attributes. 

It  should  be  noted  that  the  consistency  and  quality  of  the  DoDAF  artifacts  constructed  by  DoD  programs 
reflects  the  evolving  nature  of  architecture  as  a  discipline,  and  therefore  the  programs  with  the  best  ISP 
artifacts  are  those  that  have  not  yet  completed  development.  For  these  programs,  the  full  story  of  development 


The  views  are  characterized  as  AV  (All  View),  SV  (Systems  View),  OV  (Operational  View),  and  TV  (Technical  View),  with  a 
numerical  designator  that  corresponds  generally  to  the  level  of  detail  revealed  in  that  view  [DoDAF  2007], 
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cost  growth  has  not  been  told;  therefore,  this  analysis  method  does  not  yet  address  critical  issues  such  as 
propensity  for  cost  or  schedule  growth  relative  to  the  milestone  B  baseline.  This  deficiency  will  resolve  itself 
in  due  course,  as  programs  “age  into”  milestone  C,  the  development  budgets  will  become  sunk  costs,  and  at 
that  point,  this  method  will  be  able  to  address  development  cost  growth,  schedule  delay,  and  baseline  breaches. 

6.3  Method 

In  this  study,  we  used  the  Operational  View  2  (Operational  Node  Connectivity  Description)  and  the  Systems  View  6 
(Information  Exchange  Matrix),  as  defined  by  DoD  Architecture  Framework  Version  1.5  [DoDAF  2007]. 

6.3.1  Measuring  Constructive  Interdependence 

In  this  study,  constructive  interdependencies  are  characterized  as  information  flows  between  elements  of  the  MDAP 
systems  under  construction.  We  employed  network  analysis  techniques  to  analyze  the  information  provided  by  the 
SV-6  DoDAF  data.  Therefore,  we  equate  the  term  “node”  for  a  program  entity  and  “link”  to  represent  information 
flow.  The  use  of  the  terms  “links  and  nodes”  does  not  suggest  a  particular  implementation  of  information  flow.12 
The  means  of  information  flow  are  determined  by  a  specific  program  and  vary  considerably. 


For  this  study,  characteristics  of  constructive  interdependence  are  described  in  Table  7. 
Table  7:  Description  of  Network  Terms  Used  for  Analysis 


Item 

Symbol 

Description 

Node 

N 

An  architecture  element  that  produces, 
consumes,  or  processes  data. 

Send  node 

Ns 

A  node  that  sends  information 

Receive  node 

Nr 

A  node  that  receives  information 

Send/receive  node 

Ns/r 

A  node  that  sends  and  receives  information 

Total  nodes 

nt 

Total  number  of  nodes  within  the  system 

Fink 

F 

A  physical  or  logical  connection  between 
nodes 

Uni -directional  link 

Lud 

A  link  with  a  uni-directional  information  flow 

Bi-directional  link 

Lbd 

Fink  with  a  bi-directional  information  flow 

Total  links 

Lt 

Total  number  of  links  in  the  system 

Maximum  links 

l^Max 

Maximum  number  of  links  that  are  possible1 3 

Finks  per  node 

Ly/  Nt 

Total  number  of  links  divided  by  total 

number  of  nodes 

The  basis  for  the  definitions  of  nodes  and  links  is  the  DoD  Architecture  Framework  Version  1 .5  Volume  I:  Definitions  and 
Guidelines  [DoDAF  2007], 

13  The  maximum  possible  links  is  computed  according  to  Metcalfe's  Law,  which  states  that  the  maximum  number  of  possible 
linkages  in  a  network  is  derived  from  the  formula  LMax  =  NT  *  (NT  -  1)/2  [Metcalfe  1995]. 
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Item 

Symbol 

Description 

Integration  density 

Lt/  LMax 

Total  number  of  links  divided  by  the 
maximum  number  of  links  that  are  possible 

To  quantitatively  assess  the  number  of  nodes  and  links  within  a  system,  each  node  (entity)  was  counted  once 
and  each  link  (relationship)  was  counted  once  regardless  of  how  many  times  the  entity  (node)  or  relationship 
(link)  was  invoked  within  the  DoDAF  SV-6.  Therefore,  the  measure  of  interdependence  used  within  this  study 
is  based  on  unique  links  and  nodes.  Figure  10  provides  an  example  to  illustrate  the  counting  rules  that  were 
used  within  this  analysis. 


Actual  link 
Potential  link 


Item 

Symbol 

Value 

Send/Receive 

Ns/r 

3 

l/> 

( D 

~o 

Send 

Ns 

0 

o 

z 

Receive 

Nr 

1 

Total 

Nt 

4 

Uni-directional 

Lud 

2 

Bi-directional 

Lbd 

3 

c 

Total 

Lt 

5 

Metcalfe  number 

LtMax 

6 

Integration  density 

Lt/LtMax 

5/6 

Links  per  node 

Lt/Nt 

5/4 

Figure  1 0:  Example:  Node  and  Link  Counting  Rules  Applied  to  ISP  Documents 

6.3.2  Measuring  Development  Resource  Demand 

For  this  study,  development  resource  demand  is  defined  as  the  Research,  Development,  Test,  and  Evaluation 
(RDT&E)  program  budget. 

In  this  study,  the  number  of  links  and  nodes  represented  the  independent  variable  that  we  used  as  a  proxy 
measure  for  interdependence.  The  dependent  variable  is  development  resource  demand. 

6.4  Results  and  Discussion 

6.4.1  Part  I  -  Relationship  between  Total  Number  of  System  Nodes  and  Integration  Density 

As  we  applied  the  counting  rules  to  the  ISP  data  for  the  MDAPs  in  our  sample,  we  discovered  a  useful  pattern 
in  the  relationship  between  the  total  number  of  system  nodes  (NT)  and  the  integration  density  (L T/  LMax).  This 
pattern  is  illustrated  in  Figure  1 1 .  The  pattern  was  consistent  across  a  wide  range  of  system  sizes  and  a 
diversity  of  MDAPs  that  were  included  in  the  data  set. 
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The  consistency  of  the  pattern  provided  the  ability  to  address  variability  in  the  way  the  SV-6  DoDAF  data  was 
interpreted  from  program  to  program  and  to  objectively  address  many  of  the  apparent  inconsistencies  among 
the  documents.  For  example,  if  a  program  suggested  that  it  either  had  significantly  greater  or  fewer  links  for  a 
given  size  (that  is,  total  number  of  nodes),  a  review  of  the  supporting  documentation  would  typically  reveal 
some  fundamental  error  within  the  ISP  documentation.  Likewise,  we  noted  that  progressive  programs  that  used 
architecture  tools  tended  to  have  more  consistent  and  reliable  ISP  products. 


Integration  Density 


Figure  1 1:  Integration  Density  of  MDAPs 

Beyond  using  this  relationship  to  validate  extracted  DoDAF  data,  the  relationship  illustrated  in  Figure  1 1  also 
provides  a  way  to  estimate  the  number  of  links  that  a  program  with  a  given  number  of  nodes  would  have. 

Thus,  the  number  of  program  nodes  or  links  can  be  used  as  a  proxy  measure  for  system  size. 

6.4.2  Part  II  -  Relationship  Between  Constructive  Interdependence  and  Development  Resource 
Demand 

If  the  effect  of  complexity  and  interdependence  is  to  increase  development  effort,  we  would  expect  to  see  a 
positive  correlation  between  the  measures  of  interdependence  and  the  corresponding  RDT&E  budgets.  Stated 
more  formally  in  terms  of  the  data:  according  to  our  hypothesis,  RDT&E  resources  are  influenced  by  three 
factors: 

1 .  Number  of  nodes  and  links 

2.  Node  and  link  complexity 

3.  Interdependence-related  complexity  (measured  in  number  of  links  per  node) 

We  observed  a  positive,  non-linear  correlation  between  the  number  of  nodes  and  development  resource 
demand  as  measured  by  RDT&E  budget  (see  Figure  12)  and  also  between  the  number  of  links  and  RDT&E 
budget  (see  Figure  13).  As  size  (number  of  nodes)  and  interdependence  (number  of  links)  increase,  the  amount 
of  development  resources  required  increases  according  to  a  power-law  relationship. 
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RDT&E  $M  versus  Nodes 


Figure  12:  Development  Resource  Demand  Versus  Number  of  Nodes 
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Figure  13:  Development  Resource  Demand  Versus  Number  of  Links 
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6.4.3  Part  III  -  Development  of  the  Equivalent  Node  Measure 


To  explore  the  relationship  between  nodes,  links,  and  development  resource  demand,  a  model  was  proposed 
that  incorporates  the  combined  effects  of  size  and  interdependence.  To  develop  this  model,  we  drew  a 
conceptual  analogy  between  the  software  cost  analysis  approach  of  counting  Equivalent  Source  Lines  of  Code 
(ESLOC)  based  on  a  weighted  combination  of  new,  reused,  and  modified  software  lines  of  code  and  then  using 
this  value  as  input  to  a  size-driven  parametric  software  cost  models.14 


In  our  case,  we  explored  the  notion  of  an  equivalent  node  (which  is  a  weighted  combination  of  send-only, 
receive -only,  and  send-receive  nodes)  and  an  interdependence-driven  complexity  factor  represented  by  the 
normalized  number  of  links  per  node.  Our  proposed  model  captures  the  following  characteristics: 


1 .  Send/Receive  nodes  are  more  complex  than  Send-Only  or  Receive-Only  nodes 

2.  Send-Only  nodes  are  more  complex  than  Receive-Only  nodes 

3.  Nodes  that  have  more  links  are  more  complex  than  nodes  with  fewer  links 


The  resulting  formulation  is  expressed  by  Equation  1 . 


Ne  —  (dNs,  +  gNs  +  hNr) 


Lt / 

/A 


avg 


(l7a 


where: 


[1] 


Ne  is  the  equivalent  nodes  value 
Ns  is  the  number  of  send-only  nodes  for  the  program 
Nr  is  the  number  of  receive -only  nodes  for  the  program 

Ns/r  is  the  number  of  send-receive  nodes  for  the  program 

Lt  is  the  total  number  of  links  for  the  program 

Nt  is  the  total  number  of  nodes  for  the  program 

d  is  the  derived  coefficient  reflecting  the  relative  weight  of  the  send-receive  nodes 

g  is  the  derived  coefficient  reflecting  the  relative  weight  of  the  send-only  nodes 

h  is  the  derived  coefficient  reflecting  the  relative  weight  of  the  receive -only  nodes 

c  is  the  derived  exponent  that  scales  node  complexity  according  to  the  interdependence-driven 

complexity  of  the  program 


Most  parametric  software  cost  models  use  a  size-driven  algorithm  to  compute  cost,  effort,  and  schedule.  In  order  to  adjust 
these  models  to  predict  the  cost  of  software  that  is  reused  or  modified  rather  than  designed  from  scratch,  various  weighting 
schemes  have  been  developed.  These  weighting  methods  typically  assign  newly-developed  code  a  weighting  factor  of  1 .0, 
with  reused  or  modified  code  having  some  fraction  of  the  new  code  weight.  The  weights  calculate  the  amount  of  equivalent 
new  software  that  would  be  developed  to  have  the  same  cost  as  the  modified  and  reused  code.  Thus,  an  “equivalent”  source 
line  of  code  is  input  into  the  conventional  parametric  model. 
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Once  the  value  of  equivalent  nodes  (Ne)  is  computed,  the  relationship  between  equivalent  nodes  and 
development  resource  demand  can  be  expressed  as  in  Equation  2. 

RDT&ES  =  aNg  [2] 


where: 


RDT&ES  is  the  development  resource  demand,  monetized  as  research,  development,  test,  and 
evaluation  dollars  expressed  in  fiscal  year  2008  constant  dollars  X  106 

a  is  the  derived  scaling  coeffient 

Ne  is  the  value  for  equivalent  nodes  (calculated  by  Equation  1) 

b  is  the  derived  exponential  scaling  factor 

Having  proposed  a  functional  form  for  equivalent  nodes  (Ne)  and  development  resource  demand  ( RDT&ES ), 
the  values  of  the  various  weight  factors  and  coefficients  are  derived  using  the  ISP  data  and  an  optimization 
routine  (Microsoft  Excel  Solver)  that  minimizes  the  residuals  for  a  nonlinear  equation  relating  nodes  to 
resource  demand  (millions  of  RDT&E  dollars). 


The  results  of  the  optimization  routine  lead  to  the  following  expression: 

1.22 


Ne  =  (Ns/r  +  O.SNs  +  0.29  Nr) 


Lt/, 


Nt 


1.02 


[3] 


where  we  find  that 


d  the  relative  weight  of  send/receive  nodes,  is  set  to  1.0  (by  definition) 

g  the  relative  weight  of  send-only  nodes  is  0.5,  which  implies  that  send-only  nodes  are 

half  as  complex  as  send/receive  nodes 


h 


avg 


the  relative  weight  of  receive-only  nodes  is  0.29,  meaning  that  receive -only  nodes  are 
29%  as  complex  as  send/receive  nodes 

the  interdependence-driven  complexity  scaling  factor  is  1 .22,  meaning  that  node 
complexity  has  a  positive,  non-linear  effect  on  development  resource  demand 


Deriving  the  parameters  of  the  basic  nonlinear  equation  relating  equivalent  nodes  to  overall  development 
resource  demand  (Equation  2,  above)  results  in  the  following  expression: 

RDT&ES  =  20.7  tVe138  [4] 

where  we  find  that 

a  the  derived  scaling  coefficient,  is  $20.7  (X106) 

b  the  derived  exponential  scaling  factor,  is  1.38 


As  illustrated  in  Figure  14,  the  relationship  expressed  in  Equation  4  demonstrates  a  remarkable  fit  to  the  data 
set,  demonstrating  a  coefficient  of  correlation  of  99  percent. 
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Figure  14:  Fitted  Equation  of  Equivalent  Nodes  to  Development  Resource  Demand 

The  utility  of  the  relationship  expressed  in  Figure  14  is  significant  for  assessing  the  approximate  level  of 
development  resource  demand  for  programs  of  a  given  size  and  level  of  interdependence.  Measures  of 
constructive  interdependence  can  be  assessed  from  standard  documents  prepared  relatively  early  in  the 
program  lifecycle.  Analyzing  the  degree  of  interdependence  early  in  the  program  planning  process  will 
improve  estimates  of  development  budgets  to  ensure  that  adequate  resources  are  made  available  to  support 
systems  engineering,  architecture  development,  and  risk  mitigation  activities  during  system  development.  This 
may  reduce  the  incidence  of  program  overrun  since  the  previously  hidden  effects  of  interdependence-driven 
complexity  can  now  be  explicitly  described  and  their  effects  on  cost  escalation  predicted.  Flowever,  this  will 
also  raise  the  apparent  cost  of  programs  early  in  the  program  lifecycle  and  make  some  programs  appear 
unaffordable.  Therefore,  such  assessments  may  be  met  with  skepticism  or  outright  hostility.  When  this  is  the 
case,  the  tendency  might  be  to  defer  such  assessments  to  later  phases  in  the  development  lifecycle  so  that  these 
additional  costs  and  impact  to  schedule  are  underemphasized.  Unfortunately,  the  history  of  software  and 
systems  engineering  programs  is  replete  with  examples  of  such  a  strategy’s  enormous  impact  on  cost  growth  or 
a  dramatic  reduction  of  technical  performance. 

The  results  of  this  study  are  interesting  when  combined  with  the  results  of  Study  4.  Figure  6  and  Figure  7  show 
evidence  of  programs  sharing  resources  across  their  boundaries  and  creating  increased  financial 
(programmatic)  interdependencies  as  a  result.  Figure  13  and  Figure  14  indicate  significant  increases  in 
development  cost  demand  with  increasing  interdependence.  This  may  suggest  that  programs  facing 
overwhelming  complexity  and  associated  costs  are  searching  out  ways  to  offset  the  cost  of  this  complexity  and 
are  doing  so  by  creating  collaborative  programmatic  clusters  to  work  through  shared  challenges.  Such 
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collaboration,  while  aligned  with  the  DoD  leadership’s  promotion  of  a  joint  doctrine,  no  doubt  faces 
substantial  governance  barriers  structured  by  the  existing  service-centric  and  program-centric  paradigm. 

The  observed  power  law  relationship  between  nodes  and  their  links  also  suggests  an  underlying  innate 
phenomenon  may  be  at  work.  In  naturally-occurring  networks,  growth  occurs  by  establishing  links  among 
nodes  in  the  most  cost-effective  way  possible:  by  providing  access  to  the  greatest  number  of  resources  through 
the  fewest  possible  links.  Under  such  circumstances,  the  law  of  preferential  attachment  emerges  which  states 
that  when  adding  nodes  to  a  network,  some  nodes  are  more  attractive  targets  for  links  than  others.  These  nodes 
may  themselves  have  a  richer  set  of  resources  or  may  be  better-connected  than  others.  Therefore,  they  are  able 
to  indirectly  provide  access  to  a  wider  range  of  resources.  These  well-connected  nodes  tend  to  attract  yet  more 
connections,  becoming  highly-connected  “hubs.”  In  this  way,  through  the  natural  behavior  of  each  node 
making  local  optimizing  decisions  during  network  growth,  some  nodes  may  develop  many  links  while  the 
majority  of  nodes  might  have  only  one.  This  gives  rise  to  a  links  per  node  distribution  that  follows  a  power  law 
relationship  similar  to  that  observed  in  our  data  set. 

Such  networks  have  been  called  “scale-free”  and  manifest  interesting  and  useful  characteristics  that  in  the 
future  may  help  explain  and  predict  the  behaviors  of  the  defense  acquisition  enterprise  [Laszlo-Barabasi  2002, 
pp.  86-87].  However,  in  the  near-term,  even  these  early  and  relatively  simple  insights  provide  a  significant 
increase  in  descriptive  and  predictive  power  over  traditional  program-centric  management  approaches. 
Additional  research  in  these  areas  should  prove  extremely  fruitful. 
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7  Summary 


This  report  described  five  studies  that  were  part  of  a  major  research  initiative  sponsored  by  the  Office  of  the 
Deputy  Under  Secretary  of  Defense  (Acquisition  and  Technology)  Systems  and  Software  Engineering  / 
Strategic  Initiatives. 

The  overall  goal  of  the  research  was  to  identify,  quantify,  and  assess  the  degree  of  interdependence  and  to 
assess  the  effects  of  that  interdependence  on  program  risk. 

Throughout  all  of  these  studies,  the  researchers  attempted  to  identify  representations  within  data  constructs  that 
would  capture  the  phenomenon  of  interdependency.  However,  there  is  a  need  to  validate  the  effectiveness  of 
the  measures  through  additional  studies.  Generally  speaking,  we  do  not  want  to  assert  that  the  approaches  used 
in  our  studies  represent  the  only  ways  to  characterize  interdependency.  The  identification  of  effective  ways  of 
measuring  interdependency  is  a  topic  that  would  benefit  from  additional  research. 

Our  research  approach  began  with  Study  1  that  explored  the  qualitative  factors  that  confound  program  cost  and 
schedule  estimation.  Study  2  employed  data-mining  and  statistical  analyses  to  determine  whether  DAES- 
SARS  information  can  be  used  to  forecast  program  performance.  An  interesting  result  from  this  study  is  that 
there  was  no  evidence  that  such  indicators  are  effective  in  predicting  program  breaches.  Studies  3  through  5 
employed  network  analysis  techniques  to  quantitatively  characterize  programmatic  and  constructive 
interdependences  in  the  acquisition  enterprise.  These  last  three  studies  culminated  in  graphical  models  that 
related  interdependence  and  program  cost. 

In  this  paper,  we  reported  a  number  of  important  findings  and  noteworthy  insights  that  are  available  when 
considering  programs  in  light  of  their  interdependencies  with  other  programs.  In  particular,  four  critical 
findings  were  revealed  by  this  research. 

First,  our  research  study  found  no  evidence  that  indicators  reported  within  Defense  Acquisition  Executive 
Summary  (DAES)  reports  or  Select  Acquisition  Reports  (SARs)  predict  program  breach  events. 

Second,  limiting  the  definition  of  interdependence  to  programs  that  are  identified  and  funded  as  joint  programs 
is  insufficient.  All  programs  are  interdependent  to  some  degree,  and  therefore  interdependence-related  risk  is 
widespread,  regardless  of  service  affiliation  or  participation.  The  examination  of  individual  program 
characteristics  in  isolation  from  other  programs  falls  short  of  the  mark  for  understanding  risks  to  the  program, 
particularly  those  risks  arising  from  interdependencies.  An  expanded  definition  of  interdependencies,  based  on 
rigorous  investigation,  is  likely  to  provide  further  insights  into  program  risk. 

Third,  even  the  relatively  aggregated  SARs  data  and  the  often-disparaged  DoDAF  artifacts,  when  combined  in 
an  analytically  rigorous  manner  over  a  sufficiently  large  sample,  can  provide  significant  insights  into  program 
behaviors  that  are  unavailable  elsewhere.  Continued  investment  in  the  gathering  of  objective,  authoritative  data 
should  be  a  major  emphasis  across  the  DoD.  The  consequence  of  failing  to  gather  and  analyze  these  important 
data  is  substantial  program  failure,  wasted  resources,  and  erosion  of  the  public  trust  in  the  integrity  and 
capability  of  the  DoD. 
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Finally,  traditional  methods  of  analyzing  risk,  while  important,  need  to  be  supplemented  with  network  analysis 
techniques  to  reveal  the  true  scope  and  effects  of  programmatic  and  constructive  interdependence.  Additional 
investigation  into  methods  and  measures  that  can  reveal  critical  interdependencies  is  clearly  warranted. 

These  results  indicate  that  an  expanded  definition  of  interdependencies  along  with  the  incorporation  of 
network  analysis  tools  may  provide  important  insights  into  program  performance  in  a  joint  capability  arena.  It 
is,  thus,  an  important  topic  of  inquiry. 
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Appendix  A:  Diagnostic  Risk  Indicators  Associated  with  Integration 
and  Interoperability 


Overview 

This  appendix  presents  the  results  of  Study  1,  which  is  described  on  page  5. 

Missing  Requirements 

Missing  requirements  are  a  significant  source  of  estimation  error  and  cost  variance  related  to  interoperability 
and  integration  (I&I)  efforts.  Although  missing  requirements  have  always  troubled  complex  software  systems, 
the  issue  escalates  with  each  dimension  of  I&I  complexity:  systems,  services,  knowledge  domains,  funding 
sources,  users,  stakeholders,  and  interfaces. 


1 .  What  evidence  exists  that  the  program  has  an  understanding  of  the  complexity  of  the  problem  space,  the 
solution  space,  and  the  required  software? 


Architectural  views 

Operational 

(+) 

System 

(+) 

Technical 

(+) 

Cross-correlated15 

(++) 

Software 

(+) 

None 

(-) 

Scenario  thread  analysis 

Extensive 

(+) 

Cross  domains 

(+) 

None 

(-) 

Use  cases 

Extensive 

(+) 

User-validated 

(+) 

None 

(-) 

Stakeholder  involvement 

Continuous 

(+) 

Comprehensive 

(+) 

Measured,  common, 
shared  understanding 

(+) 

None 

(-) 

Simulation  efforts 

Extensive 

(++) 

Sparse 

(+) 

None 

(-) 

Fragmented  understanding 

Measured  assessment 

(+) 

No  evaluation 

(-) 

15  The  DoDAF  views  do  not  typically  represent  software  architectures;  therefore,  a  mapping  of  the  DoDAF  views  into  software 
architectures  should  be  evidenced. 
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2.  Requirements  volatility  is  considered  to  be  a  major  source  of  risk  to  the  management  of  large  and 
complex  software  projects. 


Team  conflict 

Minimalist  reduction  solutions  (LCD) 

Optimized  union  of  stakeholder  demands 

Measured,  common,  shared  understanding 

Clear  prioritization  of  requirements 

(+) 

(++) 

(+) 

(+) 

Volatility  management 

Claims  of  stability16 

(-) 

Volatility  acknowledged 

(+) 

Process  to  monitor 

(+) 

Formal  change  management 

(+) 

Configuration  control 

(+) 

Quality  Management  processes 

(+) 

Nature  and  origins  of  requirements  change  analysis 

(+) 

Data  models 

Simplistic  "post  all  data  to  shared  space"17 

(-) 

Holistic  DOTMLPF17  examination 

(+) 

Full  mission  thread  simulations 

(+) 

Scenario-based  vetting  of  the  system  complexities 

(+) 

Scope  creep 

Problem  solving  process  elicits  critical  requirements 

(+) 

Precisely  articulated  requirements 

(+) 

Opportunistic  goal-seeking  behaviour 

(-) 

Controls  for  unexpected  changes  in  the  operational 
environmental  context 

(+) 

Evidence  of  shared  understanding  and  consensus 

(+) 

16  Claims  that  requirements  are  stable  or  frozen  should  be  treated  with  healthy  skepticism;  true  joint  efforts  typically  exhibit  very 
volatile  requirements. 

17  Making  data  available  should  not  be  accompanied  by  an  assumption  of  utilization.  It  takes  Doctrine,  Operations,  Logistics, 
Training,  Leadership,  Personnel,  and  Facilities  (DOT-LPF)  guidance  development  to  leverage  the  capabilities  properly.  The 
budgeting  of  this  enabling  guidance  development  and  subsequent  policy  implementation  is  often  reported  to  be  lacking  in  l&l 
intensive  programs. 
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Organizational  &  Institutional  Obstacles 

Joint  teams  suffer  the  additional  complications  of  serving  many  masters.  Each  stakeholder  commonly  will 
have  separate  external  influences:  financial  and  philosophical,  if  not  statutory,  in  nature.  These  issues  generate 
inter-  and  intra-team  dynamics  that  are  unique  to  I&I  efforts. 

3.  To  what  extent  do  the  efforts  involve  formal  plans  (e.g.,  IPT,  identified  champion)  for  building  and 
maintaining  trust  within  and  among  the  various  teams? 


Team  interaction 

Team  able  to  produce  joint  artifacts 

(+) 

Teams  are  newly  formed 

(-) 

Complete  &  ongoing  participation  by 
members 

(+) 

Identified  champion  of  joint  goals 

(+) 

Veto  power 

Independent  budget  control 

(-) 

Able  to  withdraw  funding 

(-) 

0-3  independent  parties 

(+) 

Greater  than  5  independent  parties 

(-) 

Conflict  reduction 

Risk  mitigation  plans/strategies  exist 

(+) 

Formal  negotiation  practices 

(+) 

No  conflict  resolution  strategies 

(-) 

Arbitration  process 

Defined  and  agreed  to  among  all  the  key 
teams 

(+) 

Formal  process  with  lead  contractor  only 

(-) 

Each  critical  team  that  works  on  the  project 
has  established  and  agreed  on  a  formal 
process 

(+) 

No  formal  arbitration  process 

(-) 

Formal  governance 

Penalties  for  early  termination 

(+) 

Binding  agreements  among  all  stakeholders 

(+) 

No  penalties 

(-) 

Incentives 

Cut  across  organizational  boundaries 

(++) 

Money  flows  from  the  level  that  has  the 
joint  l&l  motivation 

(+) 

Based  in  separate  organization's  goals 

(-) 
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Lifecycle  Sustainment 

Lifecycle  sustainment  in  stand-alone  software  systems  is  traditionally  low  risk.  However,  the 
interdependencies  of  highly  integrated  and  interoperable  systems  do  generate  sustainment  issues — particularly 
if  constituent  parts  must  be  independently  maintained.  Transferring  these  systems  from  development  to 
operations  is  more  difficult  due  to  the  need  to  continuously  maintain  I&I  independencies. 

4.  Look  for  evidence  that  I&I  sustainment  activities  have  been  considered. 


Interdependence 

Dependencies  are  documented 

(+) 

Interactions  are  sequential  in  nature 

(++) 

Interactions  are  pooled 

(+) 

Interactions  require  reciprocal  actions 

(~) 

Well  developed  relationship  models 

(+) 

Critical  functions  are  loosely18  coupled 

(+) 

Critical  functions  are  tightly  coupled 

(-) 

DOTMLPF19 

Ongoing  sustainment  budgeted 

(+) 

Implementation  of  changes  budgeted 

(+) 

Stakeholder  Measured  CSU 

(+) 

Semantics 

Documented 

(+) 

Training  to  institutionalize  changes 

(+) 

Measured  Common  Shared  Understanding 

(+) 

Information  processes 

Historically  proven 

(+) 

Performance  measures  in  place 

(+) 

Cross  DOTMLPF  spectrum 

(+) 

Precedence 

No  mission  critical  operational  experience 

(-) 

Non-mission  critical  experience 

(+) 

Demonstrated  stability  in  a  mission  critical 
operational  setting 

(++) 

5.  Unfortunately,  the  state  of  practice  relies  heavily  on  testing  for  accomplishing  I&I.  This  leads  to 
significantly  larger  testing  budget  requirements  as  the  test-rework  loops  perform  “brute  force”  I&I. 


18  Loose  coupling  often  leads  to  shortcomings  in  security  policies,  legacy  systems  utilization,  and  complex  code  requirements. 
Has  the  program  anticipated  these  costs? 

19  DOTMLPF  is  doctrine,  organization,  training,  materiel,  leadership  and  education,  personnel  and  facilities. 
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Team  Performance 


I&I  programs  require  exceptional  team  performance  in  the  face  of  exceptional  team  composition.  Team 
members  often  come  from  disparate  organizations  with  conflicting  goals,  independent  funding,  and  localized 
incentives.  It  takes  tremendous  leadership  and  individual  commitment  and  flexibility  to  achieve  synergistic 
outcomes  in  such  environments. 

6.  The  social  constructive  nature  of  establishing  I&I  requirements  requires  highly  mature  problem  solving 
and  coordinating  efforts. 


Problem  space 

Documented,  clear  understanding 

(+) 

Measured  consensus (CSU) 

(+) 

Holistic  representation20 

(+) 

Demonstrated  transference 

(+) 

Experience 

Team  has  proven  track  record 

(+) 

Constituents  performed  in  similar  SoS 
situations 

(+) 

Formalized  coordination 

Clerical  assistance 

(+) 

Dedicated  program  champion 

(+) 

Conscientious  plan  to  monitor  the 
effectiveness  of  tools  and  techniques  for 
eliminating  time  and  distance  problems 

(+) 

Decision  behaviours 

Snap  judgements 

(-) 

Leaps  of  faith 

(-) 

Risk  aversion 

(-) 

7.  Integration  and  Interoperability  (I&I)  programs  may  demand  synthesis  of  several  domains  of  knowledge. 


Semantics 

Documented 

(+) 

Measured  consensus 

(+) 

Culture 

Unvoiced  opinion 

(-) 

Rank  disparity 

(-) 

Multiple  Operation  Specialties 

(-) 

Disparate  demands 

Measured  Common  Shared  Understanding 

(+) 

Intersection21 

(-) 

Union20 

(+) 

20  Consensus  is  responsive  to  all  the  needs  of  the  constituents,  not  a  subset. 

21  These  are  minimal,  easier,  least-common-denominator  solutions. 


41  |  CMU/SEI-2010-TR-024 


42  |  CMU/SEI-201 0-TR-024 


Appendix  B:  Additional  Analyses  Using  DAES-SARS  Information  for 
Forecasting  Performance 


Overview 

This  appendix  presents  additional  analyses  associated  with  Study  2  that  is  described  on  page  7. 

An  examination  of  other  statistical  techniques  did  not  yield  any  better  insight  than  the  Assessment  Indicators 
that  are  listed  in  Table  2  on  page  9. 

The  Earned  Value  Indicator,  for  instance,  was  greatly  variable  until  the  end  of  each  phase. 


CONTRACT  SPiyCPI 

MIDS  Boeing  Company 

F/A-1 8  INTEGRATION  -  N0001 9-91 -G-0091  (1 )  (CPFF)  As  of  02/29/2000 


SPI 

. V . 

100.00 

CPI 

- A - 

97.04 

Figure  15:  Sample  Earned  Value  Management  Report 

Text  Analysis  of  Program  Review  Reports 

The  use  of  data  mining  tools  to  extract  information  was  also  explored  due  to  the  quantity  of  textual  information 
in  the  DAES/SARs.  The  hope  was  that  such  tools  could  reduce  the  reading  load  by  quickly  displaying  the 
important  conceptual  structure  embodied  by  several  years  of  DAES/SARs.  Although  we  determined  that  the 
tools  were  not  adequate  for  total  reliance,  the  results  did  confirm  the  evolution  of  testing  issues  into  corrective 
actions  observed  by  our  reading  and  thus  served  as  a  confirmation  of  this  complex  issue.  Otherwise,  we  found 
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little  evidence  that  the  executive  DAES  reports  contained  significant  references  to  the  types  of  SoS  risk 
factors22  identified  from  the  subject  matter  expert  and  literature  reviews  conducted  in  the  early  stages  of  this 
research.  None  of  these  factors  are  systematically  captured  by  current  acquisition  reporting  requirements. 


22  These  factors  included  complexity  comprehension,  volatility  tolerance,  trust,  sustainability,  coordination  maturity,  and  domain 
coverage. 
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